1. Field of the Invention
The present invention relates to a technique of obtaining the position and orientation information of an image sensing device using an image sensed by the image sensing device.
2. Description of the Related Art
There is a method of causing an image sensing device to sense a marker arranged at a known position in a coordinate system (reference coordinate system) provided as a reference in a three-dimensional space, and obtaining the position and orientation information of the image sensing device in the reference coordinate system in real time using the feature point coordinate values of a projected image in the sensed marker image.
A method disclosed in, e.g., non-patent reference 1 (see below) corresponds to this. The method disclosed in non-patent reference 1 uses a two-dimensional matrix code (marker) which is formed from a black frame and a black-and-white rectangular matrix including a two-dimensional barcode portion. In an image obtained by causing an image sensing device to sense the marker, the image coordinate values of four feature points of the black frame and the ID of the two-dimensional barcode portion are recognized. Furthermore, the relative position and orientation between the marker and the image sensing device is obtained, thereby estimating the position and orientation of the image sensing device in the reference coordinate system.
Another method is disclosed in non-patent reference 2 (see below), in which instead of obtaining a position and orientation based on only a sensed image, an image sensing device having an orientation sensor is used to sense two or more circular markers each containing a two-dimensional code, thereby obtaining the position and orientation of the image sensing device. This is a hybrid image sensing device position and orientation estimation method.
Alternatively, as disclosed in patent reference 1 (see below), a set, which has no outer shape, of points and patterns is used as a marker. In an image obtained by sensing the marker, four feature points combined with the patterns and the ID of the marker are recognized. Additionally, the relative position and orientation between the marker and the image sensing device is obtained, thereby obtaining the position and orientation of the image sensing device in the reference coordinate system.
In a still another method practiced in general, an image sensing device itself is fixed at a known position in real space. The image sensing device captures a marker fixed on a measurement target object which moves in the real space, thereby obtaining the position and orientation of the measurement target object in real time in the coordinate system of the image sensing device.
These prior art references aim at obtaining the position and orientation of the image sensing device itself or the measurement target object by obtaining the “relative position and orientation between the image sensing device and the marker” so that a CG corresponding to the position and orientation is superimposed on the sensed image in real time.
These prior art references also aim at obtaining the position and orientation of the image sensing device itself based on the image information of the marker sensed by the single image sensing device. The prior art references are listed below.
[Non-patent reference 1] Jun Rekimoto, “Matrix: A Realtime Object Identification and Registration Method for Augmented Reality”, Proc. of Asia Pacific Computer Human Interaction (APCHI'98), 1998.
[Non-patent reference 2] E. Foxlin, Leonid Naimark, “VIS-Tracker: A Wearable Vision-Inertial Self Tracker”, IEEE Conference on Virtual Reality, pp. 199-206, LA, CA. (2003).
[Patent reference 1] Japanese Patent Laid-Open No. 2000-082108.
The above-described marker includes “feature points” to obtain the relative position and orientation between the image sensing device and the marker, and a “two-dimensional barcode portion (identification information)” to assist marker recognition or identify a plurality of markers.
In the process of causing one image sensing device to recognize a marker, if the relative angle between the image sensing device and the marker is large, or the distance between the image sensing device and the marker is long, it may be impossible to recognize the marker because of its small identification area in the image even when “feature points” are recognizable. If the marker cannot be recognized, the “relative position and orientation between the image sensing device and the marker” cannot be obtained. The position and orientation of the image sensing device in the reference coordinate system cannot be obtained, either. This state is called an “identification area recognition failure”.
<Causes of Identification Area Recognition Failure>
The major cause of the identification area recognition failure in the marker recognition process is the insufficient resolution of an index identification area to be used to identify the marker. For example, if the relative orientation between the image sensing device and the marker is deep, the resolution of the identification area is low. Conversely, if the relative orientation between the image sensing device and the marker is shallow, the resolution of the identification area is high. Hence, the deeper the relative orientation between the image sensing device and the marker is, the lower the marker recognition ratio in the marker recognition process can be. This is because an identification area with a low resolution causes a bit value recognition error or a read error of an adjacent bit value due to color smearing or noise upon image sensing.
Not only the relative orientation between the image sensing device and the marker but also a long distance between them, as described above, or a too small marker makes the projected image of the marker in the sensed image small, resulting in low resolution. This can cause a marker recognition failure.
When a marker whose identification area is in shadow of another object is sensed, the bit values included in the projected image of the marker in the sensed image may be different from those which should be. In this case, a read error of the bit values included in the identification area may occur. If a bit value read error occurs, marker recognition fails at a high possibility.
There are two types of marker recognition failures. One type is that the identification area cannot be recognized at all, and no marker can be registered. The other type is that the identification area is erroneously recognized as a marker of a wrong ID. The former failure will be called a “marker detection failure”, and the latter will be called a “marker recognition error”.
<Problem in Marker ID Recognition by a Plurality of Image Sensing Devices>
Assume that two fixed image sensing devices whose relative position and orientation is known sense a marker that expresses an ID by diving its area. Assume that marker recognition in the image sensed by one image sensing device succeeds, whereas marker recognition in the image sensed by the other image sensing device fails because of the “identification area recognition failure”.
Conventionally, the marker recognition process is independently done in the image obtained by each image sensing device. For this reason, when “feature points” in the image sensed by one of the image sensing devices are recognized, only the position and orientation, in the reference coordinate system, of the image sensing device which has succeeded in recognizing the marker is obtained. The position and orientation of the other image sensing device which has failed in marker recognition cannot be obtained.